Skip to content

feat!: make request-body compression configurable - #1065

Merged
vdusek merged 9 commits into
v3from
feat/http-compressors
Sep 15, 2026
Merged

vdusek merged 9 commits into
v3from
feat/http-compressors

Conversation

@vdusek

@vdusek vdusek commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

TLDR

Description

Request compression was hard-wired in maybeCompressValue(): brotli at quality 6, a silent fallback to gzip, and no way to pick the algorithm, tune the quality, or plug in a custom compressor.

The client now has an abstract HttpCompressor (contentEncoding plus an async compress()), a GzipHttpCompressor (quality 1 to 9, default 9) and a BrotliHttpCompressor (quality 0 to 11, default 6), all exported from the package. ApifyClient takes a compression option: 'brotli' (the default), 'gzip', or an HttpCompressor instance. The option is validated by shape, so an instance from a second copy of the package passes too. HttpClient holds the resolved compressor and the compress interceptor uses it. The rules for which bodies get compressed are unchanged: at least 1 KiB, a compressible content type, no caller-set Content-Encoding, Node.js only.

Two deliberate differences from Python. Brotli stays the default, since Node ships it in node:zlib and the dependency reason for Python's gzip default doesn't apply. Streams are never compressed, where Python reads file-likes into memory. setRecord() still has no contentEncoding option, so the docs route pre-compressed bodies through client.httpClient.call().

Docs: a new HTTP compression concepts page adapted from the Python one, a section in the v3 upgrade guide, and the regenerated api-extractor report.

Breaking change

The silent fallback from #990 is gone. The configured compressor is the only one the client runs, and its error fails the request, as in Python. A runtime whose node:zlib lacks brotli used to get a gzipped or uncompressed body and now needs compression: 'gzip'. A client constructed without the option sends the same requests as before.

✍️ Drafted by Claude Code

BREAKING CHANGE: The client runs only the configured compressor (brotli by default) and lets its error fail the request. It no longer falls back to gzip, or to an uncompressed body, when brotli fails. Pass `compression: 'gzip'` on a runtime whose `node:zlib` lacks brotli.
@vdusek vdusek added the t-tooling Issues with this label are in the ownership of the tooling team. label Sep 14, 2026
@vdusek vdusek self-assigned this Sep 14, 2026
@github-actions

github-actions Bot commented Sep 14, 2026 •

Copy link
Copy Markdown
Contributor

⚠️ There are broken links in the documentation.

See more at https://github.com/apify/apify-client-js/actions/runs/35016412232#summary-104540847859

@vdusek
vdusek marked this pull request as ready for review September 14, 2026 11:10
@vdusek
vdusek requested a review from szaganek as a code owner September 14, 2026 11:10
@vdusek
vdusek requested a review from janbuchar September 14, 2026 11:10
# Conflicts:
#	rsbuild.config.ts
#	src/http_client.ts
#	src/interceptors.ts
#	src/utils.ts
#	test/utils.test.ts

@janbuchar janbuchar left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pre-approving, thanks

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm still not convinced that this matters enough to warrant a fully-fledged guide.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's review the whole docs as part of #851 and align it with the Python version

Comment thread src/http_compressors/base.ts Outdated
* const client = new ApifyClient({ token: 'my-token', compression: new IdentityCompressor() });
* ```
*/
export abstract class HttpCompressor {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Abstract class with abstract methods only sounds like an interface.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not really, instanceof HttpCompressor works but you can't check with code if something implements an interface (without zod et al)

@janbuchar janbuchar Sep 15, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True that, but we don't really do instanceof much here, do we? I'd say that pitfalls such as breaking if you have multiple versions of apify-client in your node_modules or when crossing realms (service workers) still apply here. Typescript + a rough-shape-check via zod is preferable IMO.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changed to interface

@vdusek
vdusek merged commit a98b91f into v3 Sep 15, 2026
8 checks passed
@vdusek
vdusek deleted the feat/http-compressors branch September 15, 2026 19:57
vdusek added a commit that referenced this pull request Sep 16, 2026
The configurable request compression from #1065 moves into the shared
pipeline of the new `HttpClient` base, where the compression step lives
now that `http_client.ts` and `interceptors.ts` are gone. `compression`
becomes an `HttpClientOptions` field resolved in the base constructor, so
`ApifyClient` forwards the option instead of resolving it, and a custom
HTTP client gets the same compression as `AxiosHttpClient`.

The new HTTP clients concept page moves to `08_http-clients.md`, since
#1065 took the `07_` slot for HTTP compression.
B4nan pushed a commit that referenced this pull request Sep 30, 2026
### TLDR

- Ports the `http_compressors` design of the Python client
(apify/apify-client-python#927) to the JS client.
- Closes: #1064

### Description

Request compression was hard-wired in `maybeCompressValue()`: brotli at
quality 6, a silent fallback to gzip, and no way to pick the algorithm,
tune the quality, or plug in a custom compressor.

The client now has an abstract `HttpCompressor` (`contentEncoding` plus
an async `compress()`), a `GzipHttpCompressor` (quality 1 to 9, default
9) and a `BrotliHttpCompressor` (quality 0 to 11, default 6), all
exported from the package. `ApifyClient` takes a `compression` option:
`'brotli'` (the default), `'gzip'`, or an `HttpCompressor` instance. The
option is validated by shape, so an instance from a second copy of the
package passes too. `HttpClient` holds the resolved compressor and the
compress interceptor uses it. The rules for which bodies get compressed
are unchanged: at least 1 KiB, a compressible content type, no
caller-set `Content-Encoding`, Node.js only.

Two deliberate differences from Python. Brotli stays the default, since
Node ships it in `node:zlib` and the dependency reason for Python's gzip
default doesn't apply. Streams are never compressed, where Python reads
file-likes into memory. `setRecord()` still has no `contentEncoding`
option, so the docs route pre-compressed bodies through
`client.httpClient.call()`.

Docs: a new `HTTP compression` concepts page adapted from the Python
one, a section in the v3 upgrade guide, and the regenerated
api-extractor report.

### Breaking change

The silent fallback from #990 is gone. The configured compressor is the
only one the client runs, and its error fails the request, as in Python.
A runtime whose `node:zlib` lacks brotli used to get a gzipped or
uncompressed body and now needs `compression: 'gzip'`. A client
constructed without the option sends the same requests as before.

*✍️ Drafted by Claude Code*
janbuchar pushed a commit that referenced this pull request Nov 10, 2026
…1179)

Adds `ActorClient.server()`, which returns an `ActorServerClient` for
Server Actors (formerly Actor Standby):

```ts
const actorServerClient = client.actor('username/actor-name').server();
const response = await actorServerClient.request('/search', { params: { q: 'apify' } });
const url = await actorServerClient.getUrl();
const spec = await actorServerClient.getOpenApiSpec();
```

- `request()` sends a request to the Actor's server. The host always
comes from the Actor's `standbyUrl`. The token goes in the
`X-Apify-Authorization` header, which the platform strips, so headers
the caller passes, `Authorization` included, reach the Actor as they
are. It doesn't follow redirects and returns a redirect response as it
is, so the token can't reach a host the Actor's code picked.
- `request()` returns the server's response whatever its status. Only
429 and 503 are retried, since other statuses may come from the Actor's
code. Network errors and timeouts aren't retried, since the Actor may
have already handled the request. It throws only when it gets no
response: the Actor doesn't exist or isn't a Server Actor, or the
request fails or times out.
- `request()` sends bodies uncompressed unless `compression` is set,
returns the body parsed, as a buffer, or as a stream depending on
`responseType`, and defaults to the `long` timeout to cover a cold
start.
- `getUrl()` returns the server URL. It's fetched once per
`ActorClient`, by `getUrl()` or `getOpenApiSpec()`.
- `getOpenApiSpec()` returns the web server schema from the build that
serves the server (`actorStandby.build`, or `latest`), or `undefined` if
there's none.

`ActorClient.standby()` is a deprecated alias of `server()` and will be
removed in v4.

Other changes:

- `ApifyRequestConfig` takes new per-request options: `compression`,
`authorizationHeader`, `throwOnErrorStatus`, `retryStatusCodes`,
`retryTransportErrors`, and `followRedirects`. A custom HTTP client that
overrides `call()` should honor them, and one that overrides
`sendRequest()` must honor the new `HttpRequest.followRedirects` field,
or it would follow a redirect with the token.
- A Server Actors concepts page, unit tests, and an integration test
against a temporary echo Server Actor.

Port of apify/apify-client-python#1153.

Closes: #1159

*✍️ Drafted by Claude Code*

2.x backport: Built on the 2.x axios `HttpClient`, which gains opt-in `throwOnErrorStatus`, `retryStatusCodes`, `retryTransportErrors` and `doNotCompress` request options. There is no `compression` option, since it depends on the pluggable compressors from #1065. The request body is always sent uncompressed. The `AbortSignal` support and the timeout tiers are not ported, and errors are plain `Error`s.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

t-tooling Issues with this label are in the ownership of the tooling team.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants